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INTRODUCTION 


Problem  and  Background 

For  some  time,  the  Congress  and  the  Department  of  Defense  (DoD)  have  been 
concerned  about  the  relevance  and  utilization  of  human  resource  RDT&E  end-products. 
The  primary  focus  of  this  concern  has  been  on  the  need  to  develop  methods  designed  to 
increase  the  probability  that  the  research  products  and  outcomes  will  be  used.  To 
illustrate,  in  1977,  the  General  Accounting  Office,  after  an  extensive  review  of  tri¬ 
service  human  resources  RDT&E  laboratories,  made  the  following  recommendation:  "A 
management  monitoring  and  feedback  system  needs  to  be  developed  and  implemented. 
The  system  can  be  used  to  see  that  usage  criteria  are  applied.  .  .communications  are 
taking  place.  .  .and  problems  are  given  management  attention."1  DoD  concurred  in  this 
recommendation. 

In  response  to  this  concern,  the  Navy  Personnel  Research  and  Development  Center 
(NAVPERSRANDCEN)  included  the  following  as  goal  number  seven  of  its  corporate  5- 
year  plan  (FY81-FY85):  Develop  a  comprehensive  applications  program  to  enhance  the 
Center's  user  orientation  and  to  increase  utilization  of  the  Center's  R&D  products. 

Objective 

The  objective  of  this  effort  was  to  develop  a  user-oriented  system  for  assessing  user 
response  to  NAVPERSRANDCEN  products.  This  system  will  be  a  major  step  in  attaining 
goal  number  seven. 


APPROACH 

In  this  system,  which  was  designed  based  on  earlier  efforts  by  the  Air  Force  Human 
Resources  Laboratory,  actual  and  potential  users  are  encouraged  to  evaluate  research 
end-products.  The  requested  evaluations  are  structured  in  ways  that  describe  the 
interactions  between  R&D  producers  of  R&D  and  the  user  communities.  In  essence,  the 
system  provides  measures  of  NAVPERSRANDCEN's  responsiveness  to  user  requirements. 

During  FY  1981,  the  system  was  pilot-tested  using  a  selected  sample  of  NAVPERS¬ 
RANDCEN  technical  reports,  special  reports,  and  technical  notes  describing  Center 
products.  Commands  previously  identified  as  user  organizations  were  asked  to  review  the 
reports  selected  and  complete  a  questionnaire  designed  to  answer  the  following  questions: 

1.  Type  and  frequency  of  communication  between  the  user  and  NAVPERSRAND¬ 
CEN. 

2.  The  extent  of  user  involvement  in  phases  of  the  research,  including  planning, 
research  design,  analysis,  and  interim  or  final  recommendations. 

3.  Satisfaction  with  the  product's  timeliness,  completeness,  clarity,  relevance, 
feasibility  of  implementation,  and  projected  benefits  of  implementation. 


1  Human  resources  research  and  development  results  can  be  better  managed— 
Department  of  Defense.  Washington,  DC:  Comptroller  general's  report  to  the  Committee 
on  Appropriations,  House  of  Representatives,  April  22,  1977. 
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4.  Whether  or  not  the  user  conducted  a  formal  management  review  of  the  product. 

5.  Impact  of  the  research. 

6.  Overall  user  evaluation  of  the  research. 

The  data  base  developed  in  the  pilot  effort  demonstrated  the  feasibility  of  the 
evaluative  procedures.  Further,  the  reactions  of  both  users  and  research  managers  were 
sufficiently  positive  to  warrant  continuing  development  of  the  tracking  system.  The 
Chief  of  Naval  Operations  (OP-115)  recognized  the  potential  value  of  the  tracking  system, 
and  provided  the  support  required  to  continue  development  and  to  conduct  a  full-scale 
test  of  the  system.  This  major  effort,  which  involved  all  of  the  Center's  FY81  report 
products,  was  initiated  in  October  1981  and  completed  in  March  1982. 

The  steps  in  the  user -oriented  system  are  listed  below: 

1.  When  a  report  is  ready  for  publication,  the  principal  author  or,  if  prepared  by  a 
contractor,  the  contracting  officer's  technical  representative  (COTR)  is  asked  to  provide 
the  names,  addresses,  and  telephone  numbers  of  individuals  who  would  be  most  likely  to  be 
aware  of  the  described  research  effort  and  who  would  have  a  primary  role  in  imple¬ 
menting  the  product. 

2.  A  copy  of  the  published  report  and  the  Research  Utilization  Evaluation  Report 
(see  appendix)  are  mailed  to  the  previously  identified  users  with  a  request  for  appropriate 
action. 


3.  If  the  evaluation  report  has  not  been  returned  within  a  reasonable  length  of  time, 
a  second  request  for  evaluation  is  forwarded. 

4.  If  the  evaluation  again  is  not  returned,  telephone  contact  is  attempted  as  the 
final  follow-up  action. 

5.  Information  provided  by  the  returned  questionnaires  becomes  part  of  a  comput¬ 
erized  data  base  immediately  available  for  a  variety  of  statistical  and  graphical 
manipulations. 

In  FY81,  NAVPERSRANDCEN  published  79  reports.  User  evaluations  were  not 
requested  for  18  of  these  reports,  either  at  the  request  of  the  research  program  director 
or  because  they  were  considered  inappropriate  for  evaluation  (e.g.,  bibliographies).  For 
the  remaining  61  reports,  133  evaluation  requests  were  sent  to  individuals  representing 
the  following  organizations: 

1.  Department  of  Defense. 

2.  Office  of  Personnel  Management. 

3.  Chief  of  Naval  Operations. 

4.  Headquarters,  Marine  Corps. 

5.  Chief  of  Naval  Material. 

6.  Naval  Education  and  Training  Command. 

7.  Naval  Technical  Training  Command. 

8.  Naval  Military  Personnel  Command. 

9.  Navy  Recruiting  Command. 

10.  Office  of  Naval  Research. 

1 1.  Navy  Surface  Weapons  Center. 
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12.  NAVPERSRANDCEN.2 

13.  Center  for  Naval  Analyses. 

14.  Commander  Naval  Air  Force,  U.S.  Pacific  Fleet. 

15.  Commander  Naval  Air  Force,  U.S.  Atlantic  Fleet. 

A  total  of  99  user  evaluations  were  returned,  for  a  response  rate  of  74  percent. 
Evaluations  were  not  returned  for  eight  of  the  reports. 


RESULTS 

Overall,  79  percent  of  the  respondents  felt  that  the  amount  and  kind  of  communica¬ 
tions  with  NAVPERSRANDCEN  during  the  course  of  the  research  were 
sufficient.  Sixty-four  percent  claimed  that  they  were  actually  involved  with  some 
aspect  of  the  research  effort,  and  78  percent  felt  that  the  degree  of  involvement  with 
NAVPERSRANDCEN  was  sufficient.  When  asked  to  assess  whether  the  end  product 
would  result  in  either  cost  savings  or  increased  efficiency  or  effectiveness,  59  percent 
responded  yes,  and  45  percent  indicated  that  the  product's  implementation  potential  had 
already  been  formally  reviewed.  This  last  percentage  is  somewhat  misleading  since  some 
respondents  claimed  to  have  implemented  the  product  on  the  day  of  receipt;  therefore,  a 
formal  review  of  implementation  potential  was  not  required.  Table  1  provides  responses 
to  these  questions  by  general  category  of  user. 

A  variety  of  elements  enter  into  any  assessment  of  user  satisfaction  with  a  given 
research  product.  These  include  (1)  timeliness  of  completion,  (2)  the  completeness  of  the 
effort,  (3)  clarity  of  the  recommendations  offered,  (4)  relevance  of  the  findings  to  the 
problem,  and  (5)  feasibility  of  implementation.  This  last  element,  in  turn,  reflects  both 
the  cost  of  implementation  and  the  projected  benefits.  As  shown  in  Table  2,  there  is  a 
differential  satisfaction  with  these  elements  on  the  part  of  the  user  community. 

A  global  estimate  of  user  satisfaction  obtained  showed  that  76  percent  of  all  users 
responding  were  satisfied  with  the  research  product.  Table  3  provides  a  breakout  by  user. 

The  questionnaires  also  included  a  number  of  open-ended  questions  to  obtain  user 
comments  on  various  aspects  of  Center  research.  The  information  provided  by  such 
comments  is  extremely  valuable  as  a  diagnostic  aid,  substantially  increasing  the  quality  of 
feedback  to  Center  researchers.  The  following  comments  are  representative  of  those 
extracted  from  user  evaluations: 

1.  "It  is  anticipated  that  this  screening  vehicle  will  save  a  substantial  amount  of 
money  by  reducing  recruiter  turnover  in  the  field  due  to  failure.  More  importantly,  it 
should  increase  average  recruiter  productivity." 

2.  "It  is  recommended  that  projects  of  this  nature  be  coordinated  with  the  system 
sponsor  during  the  formulation  phase  (of  the  research)  in  order  to  obtain  more  beneficial 
data." 


2 For  several  reports,  a  code  within  NAVPERSRANDCEN  was  considered  to  be  one  of 
the  users  of  the  R&D  end  product  reported. 
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Table  1 

Percent  of  Users  Answering  "Yes”  to  Selected  Questions 


CNET  CNO  CNM 

Question  N  %  N  %  N  % 

Was  there  sufficient 
communication  with 
NAVPERSRANDCEN 
during  the 

research?  15  93  15  67  9  89 

Were  you  personally 
involved  in  any  part 
of  the  research 

project?  19  63  17  53  12  58 

Was  there  sufficient 
involvement  with 
NAVPERSRANDCEN 
during  the 

research?  15  87  16  69  11  73 

Will  the  potential 
or  actual  impact  of 
the  research  result 
in  cost  savings  or 
increased  efficiency/ 

effectiveness?  15  60  15  47  9  56 

Has  there  been  a 
formal  management 
review  on  the  imple¬ 
mentation  potential 

of  this  report?  18  44  15  53  12  42 


CNTT  NMPC  OPM  NPRDC  All  Others 
N  %  N  %  N  96  N  %  N  % 


10  90  5  100  4  75  9  100  13  33 


10  70  7  57  5  40  9  100  16  73 


9  89  5  100  4  100  9  100  11  30 


6  83  3  67  2  50  8  63  12  64 


9  56  5  60  3  30  9  44  14  29 


Table  2 


Percent  of  User  Satisfaction  with  Various 
Elements  of  NAVPERSRANDCEN  Research 


Element 

Satisfied 

Percentage 

Dissatisfied 

N 

Responses 

Relevance 

77 

23 

SI 

Completeness 

74 

26 

87 

Clarity 

73 

27 

80 

Projected  benefits 

74 

26 

79 

Timeliness 

66 

34 

85 

Implementation  feasibility 

69 

31 

80 

Projected  cost 

50 

50 

66 

Table  3 

Percent  of  Overall  Satisfaction  with 
NAVPERSRANDCEN  Research  by  User 


User 

Satisfied 

Dissatisfied 

CNET 

79 

21 

CNO 

67 

33 

CNM 

55 

44 

CNTT 

88 

12 

NMPC 

100 

0 

OPM 

75 

25 

CNRC 

100 

0 

NAVPERSRANDCEN 

100 

0 

All  others 

67 

33 

3.  .  .results  of  this  type  effort  need  to  be  identified  in  terms  of  O&MN  funding 
much  earlier.  .  .one  of  the  responsibilities  of  NAVPERSRANDCEN,  when  undertaking 
research  efforts  in  direct  support  of  fleet  operations,  is  to  forward. .  .a  financial  estimate 
of  follow-on  operational  training/training  development  necessary  to  complete  the 
tasks.  . ." 

4.  "As  user  and  sponsor  of  this  research  effort,  1  couldn't  be  more  pleased  with  the 
product." 
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CONCLUSION 


The  results  clearly  illustrate  that  the  system  has  potential  for  initiating  and 
maintaining  a  productive  dialogue  between  research  activities  and  operational  consumers. 
Data  obtained  could  serve  to  improve  the  quality  of  R&D  management  decisions,  offering 
both  long-term  trend  information  and  immediate  feedback  regarding  product  utilization 
across  a  wide  range  of  user  response  combinations.  Visualizing  the  user-oriented  tracking 
system  as  a  diagnostic  evaluation  could  provide  general  and  specific  information  on  the 
status  of  the  R3cD  program.  In  providing  a  delineation  of  problem  areas,  it  can  also 
indicate  whether  recommendations  for  corrective  action  can  be  reasonably  inferred  from 
the  data. 

Finally,  since  at  least  one  aspect  of  defining  productivity  involves  an  assessment  of 
quality  rather  than  quantity  of  production,  the  system  is  able  to  offer  management  an 
alternative  or  supplemental  productivity  measure. 


RECOMMENDATION 

It  is  recommended  that  other  Navy  R&D  laboratories  determine  the  feasibility  of 
using  this  type  of  tracking  system. 
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NAVY  PERSONNEL  RESEARCH  AND  DEVELOPMENT  CENTER  (NPRDC) 
RESEARCH  UTILIZATION  EVALUATION  REPORT 

Send  this  fora  to  NPRDC  (Code  303),  San  Diego,  CA  92152 


(D8) 

(Rl) 

(RS)  P  or  S 
(US) 


I.  Completed  by  NPRDC 
(TT)  Title  of  Product: _ 


(PN)  Technical  Report  Number: _  (PE)  Program  Element  Number: _ 

(WU)  Work  Unit  Number: _ 

(OR)  Original  Requirement  for  this  Project:___ _ 

(PM)  Contract/In-House:  _ _ 

(CO)  Originating  NPRDC  Code: _ 

(PI)  NPRDC  Principal  Investigator  for  this  Project: _ 

(IV)  Investigator  Autovon  Number: _ 

II.  Completed  by  User/Sponsor: 

(UN)  Name: _  (RK)  Rank: _ 

Last  First  MI 

(TI)  Title: _ 

(UV)  Autovon  Number: _ 

(UC)  Commercial  Phone  Number: _ 

(OS)  Office  Symbol : 

(UA)  Address: _ 

Base  or  City  State  Zip 

(TU)  Type  of  Customer: _ 1  Navy  _ 2  Army  _ 3  Air  Force 

4  Marine  Corps  _ 5  University  _ 6  Other 

(PO)  User/Sponsor's  Parent  Organization: _ 


A-l 


..  fh<<s  evaluation  form. 

The  OTSDC  utlllr.tion  “J  *"”“t1^C‘cSet'o»!SS,h  Me,o,  «  «lM' 

-  this  research  project,  about  how  often  were  each 
m.  During  ^e  course  ^  researcher_user  coanunications  used; 


(C) 

(D) 

(E) 

(F) 

Every 

Twice 

Once  a 

two 

a 

year 

months 

year 

or  less 

Never 

Weekly  Ever?  ^  Zlr 

two  d  yeat 

or  more  „.,r  orless  Never 

often  Monthly  months  2£ar_  fit  - 

(XC)  Telephone  calls?  - -  - 

CWC)  Written  Conmunications  - — 

(memos,  progress  reports, 
etc.)? 

(W)  Personal  visits?  - — 

(CW)  Conferences,  workshops?  - -  - 

indict,  your  re.oeion  to  th.  toUoutog  by  olrcUng  the  .OPtotr  , 

response  of  „s.„eh,r-us.r  eo-»nlo.rlon  ...  suftlolc.  <or 

this  project. 

(FC)  ,  researcher-user  communication. 

B.  There  should  have  been  more  researener  use 


IV.  To  what  extent  was  your  organization  involved  in  the  following  stages 
of  this  research  project: 


(Fill  in  the  blank  with  one  of  the  numbers  below) 


1. 

To 

a 

very  great  extent 

4. 

To  a 

little  extent 

7.  Don't  know 

2. 

To 

a 

great  extent 

5. 

To  a 

very  little  extent 

3. 

To 

a 

moderate  extent 

6. 

None 

(PS)  _ Planning:  Identification  of  research  needs,  defining  the  objective(s) , 

scope,  data  requirements,  etc. 

(DS)  _ Design:  Deciding  upon  methodology,  selection  of  techniques  of  data 

collection  and/or  analysis,  design  of  data  collection  instruments,  etc. 

(AS)  Analysis:  Description,  explanation,  interpretation  of  data. 

(FS)  _ Formulation  of  Recommendations:  Recommendation  of  adoption 

of  new  or  revised  programs,  policies,  procedures,  regulation, 
manuals,  equipment,  courses  of  instruction,  legislation,  etc. 


Indicate  your  reaction  to  the  following  statements  by  circling  the  appropriate 
response  (A  or  B) 

A.  For  this  research  project,  the  total  amount  of  user  organizational 
involvement  was  sufficient. 

(SS) 

B.  For  this  research  project,  there  should  have  been  more  involvement 
by  the  user  organization. 


(S8)  Were  you  personally  involved  in  any  of  the  above  stages  of  the  research 
project? 


Yes  No 


(S9)  Comments: 


A- 3 


V.  How  satisfied  are  you  with  each  of  the  following  aspects  of  this 
research: 

(Fill  in  the  blank  with  one  of  the  numbers  below) 


(ST) 

(SC) 
(SR) 
(SF) 
(SI) 
(SP) 
(SB) 

(SD) 


1.  Very  satisfied 

2.  Somewhat  satisfied 

3.  Neither  satisfied  or  dissatisfied 


Timeliness  of  response? 

Completeness  of  study? 

Clarity  of  recommendations? 

Relevance  of  findings  to  the  problem? 
Feasibility  of  implementation? 

Cost  of  implementation 

Projected  benefits  of  Implementation? 


4.  Somewhat  dissatisfied 

5.  Very  dissatisfied 


PROVIDE  ANY  INFORMATION  AVAILABLE  ON  THE  SATISFACTION  OR  DISSATISFACTION 
OF  THE  USER  WITH  THE  RESULTS  OF  THE  RESEARCH.  (Provide  documentation  if 
available;  e.g.,  letters,  memorandums,  etc.) 


VI.  Has  there  been  a  formal  or  management  review  of  the  Implementation 
potential  of  this  effort,  e.g.,  by  decision-making  Individuals  or 
committees  representing  your  Command  or  higher  echelons? 


(FF) 


No.  Why  not? 


Yes.  Please  describe  the  findings  of  this  review. 
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VII.  The  research  will  intact  as  fellows  (if  applicable): 


Place  a  "P"  next  to  each  item  the  research  has  a  potential  intact  on. 
Place  an  "A"  next  to  each  item  that  the  research  has  already  inq>acted  on. 


(OD) 

(OP) 

(01) 

(OM) 


(CP) 

(CC) 

(CR) 

(CS) 

(CD) 


(DI) 

(DR) 

(DM) 

(MP) 

(TD) 

(DO) 

(F.M) 


(SH) 

(DE) 

(HF) 

(HE) 

(DP) 


(RH) 

(RM) 

(RT) 

(RR) 

(OT) 


Operational  Commanders 
Changes  in  doctrine 
Changes  in  procedures 

Information  on  human  capabilities  and  limitations 
Modification  in  requirements  for  manpower  or  equipment 

Personnel  and  Manpower 

Changes  in  management  policy  or  techniques 

Changes  in  planning  capability 

Changes  in  manpower  requirements 

Solutions  to  specific  problems 

Information  on  which  to  base  RljD  requirements 

Training  Managers 

Development  of,  or  change  in,  course  of  instruction  or 
training  programs 

Development  of  requirements  for  training  curricula  and 
equipment 

Development  of,  or  change  in,  instructional  delivery 
methods  and  media 

Changes  in  management  policy  or  practices 
Training  device  prototypes 

Information  on  which  to  base  long  range  objectives  and 
further  R&D  requirements 

Evaluation  of  specific  materials  and  procedures 
System  Developers 

Information  on  human  capabilities  and  limitations 
Evaluation  of  specific  designs 

Human  Factors  Engineering  (HFE)  design  principles 

Efficient  ways  of  applying  HFE 

Changes  in  development  management  practices 

R§D  Community 

Information  about  human  capabilities  and  limitations 
Information  on  the  effectiveness  of  various  manpower, 
personnel,  or  training  programs 
Solutions  to  technical  problems 
Indications  of  further  R6D  requirements 

Other  (describe) 
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SPECIFIC  REFERENCES  TO  DOCUMENTATION,  IF  AVAILABLE. 


If  yes,  explain  how. 


IX.  Did  the  research  satisfy  the  user  needs?  (Circle  one  of  the  responses.) 


1.  To  a  very  great  extent 

2.  To  a  great  extent 

3.  To  a  moderate  extent 


4.  To  a  little  extent 

5.  To  a  very  little  extent 

6.  Not  at  all 


Explain  or  expand  your  response  to  the  question  above 
if  applicable. 
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X.  Where  do  you  believe  the  authority  to  inclement  the  bulk  of 
the  recommendations  (if  any)  is  located? 

A.  _ No  reconnendations  were  Hade. 

B.  _  The  operational  unit  level. 

(XI)  C.  _  The  intermediate  management  level. 

D.  _  The  major  organization  management  level  (Fleet  or 

Systems  Consnand) 

E.  _ Office  of  CNO/ Chief  of  Staff 

F.  _  Secretary  of  Defense  management  level  or  above. 

XI.  Which  of  the  following  best  describes  the  role  of  your  organization 
concerning  the  implementation  of  the  recommendations? 

A.  _ No  recommendations  were  made. 

B.  _ We  have  no  role  in  implementation. 

(X2)  C.  _  We  have  only  an  advisory  role. 

D.  _  We  participate  in  a  group,  committee  or  council  which 

decides. 

E.  _  We  have  the  authority  to  make  the  implementation  decisions. 

(X9)XII .  If  your  response  to  the  above  question  was  "no  role",  or  "only  an 
advisory  role",  who  does  have  the  authority  to  make  implementation 
decisions? 


XIII.  Which  of  the  following  best  describes  the  relationship  of  your 

organization  to  the  program  which  was  the  primary  subject  of  this 
report? 


A.  _  We  conduct  the  actual  operations  of  the  program  but  the 

overall  management  occurs  at  a  higher  level. 

B.  _  We  manage  the  program  but  the  actual  conduct  of  the 

operations  of  the  program  occurs  at  a  lower  level. 

(X3)  C.  _  We  have  the  responsibility  for  both  the  actual  conduct  of 

the  operations  and  the  overall  management  of  the  program. 

D.  _  We  neither  manage  nor  conduct  the  operations  of  the  program 

but  we  advise  either  the  managers  or  the  operations. 

(X7)  XIV. If  dollar  resources  are  required  for  implementation  and  operational 
use,  have  they  been  identified  and  added  to  the  POM  submission? 

Yes  _  No  _  Not  Required 

(UT)  XV.  Comments: 


A-7 


DISTRIBUTION  LIST 


Chief  of  Naval  Operations  (OP-115),  (OP-987H) 

Chief  of  Naval  Material  (NMAT  05) 

Commander  David  W.  Taylor  Naval  Ship  Research  and  Development  Center 
Commander  Naval  Air  Development  Center 
Commander  Naval  Ocean  Systems  Center 
Commander  Naval  Surface  Weapons  Center 


Commander  Naval  Weapons  Center 

Commanding  Officer,  Naval  Coastal  Systems  Center 

Commanding  Officer,  Naval  Underwater  Systems  Center  . .  .  . 

Commander,  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences,  ^j?**™*™ » 
Commander,  Air  Force  Human  Resources  Laboratory,  Brooks  Air  Force  Base  (AFHRL/Ai) 
Defense  Technical  Information  Center  (DDA),  (12) 
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